Monitoring and Surveillance System Arranged for Processing Video Data Associated with a Vehicle, as well as Corresponding Devices and Method

ABSTRACT

A monitoring and surveillance system arranged for processing video data associated with a vehicle, wherein said system is arranged to operate in at least two operating modi, a first modus of said two modi being associated with a first latency requirement for said video data and a second modus of said two modi being associated with a second latency requirement, said system comprising a camera unit, arranged to be installed in said vehicle, wherein said camera unit is arranged for capturing video data; a streaming unit, arranged to be installed in said vehicle, and arranged for receiving said video data and for transmitting said video data over a telecommunication network to a video processing server; said video processing server arranged for selecting a modus of said at least two operating modi, and for communicating said selected modus, over said telecommunication network, to said camera unit such that said streaming unit can be tuned to said selected modus. Complementary systems and methods are also presented herein.

FIELD OF THE INVENTION

The present disclosure generally relates to the field of a monitoringand surveillance system and, more specifically, to an assembly arrangedto be installed in a vehicle for capturing video and a video processingserver for processing the captured video.

BACKGROUND

Nowadays, trains and other vehicles may have two types of camera systemsinstalled. A front facing camera may be deployed at the front of a trainwhich front facing camera observes forwards, i.e. towards the track overwhich the train is to be steered. The front facing camera may be usedfor advanced driver assistance functions, such as obstacle on trackdetection or the like. The images of the front facing camera may also berecorded as surveillance feed which can be used later for, for example,accident examination.

Another type of camera is the surveillance camera like a CCTV system.Surveillance cameras are often mounted in wagons. Typically, twosurveillance cameras are mounted in a single wagon. When the trains has,for example, ten wagons, there are twenty or more cameras on-board. Thedefault operation would be to store the CCTV feeds on storage systemsinside the trains, and be frequently moved to a central storage system.

Surveillance cameras are typically sued to examine an incident, afterthe incident has occurred. Such incidents may be pickpocketing thefts,vandalism, or anything alike.

For self-driving cars, i.e. autonomous driving, and remote-drivenvehicles, the inventors have found that there is a desire to use theCCTV cameras for monitoring purposes to secure, that the vehicleoperations are according to plan. For monitoring purposes, it isnecessary to stream the video data of the CCTV cameras into an operationcontrol centre. One of the problems here is that the video feeds of onlysome, or at most one, camera should be streamed in order to save uplinkbandwidth, and resulting transmission costs.

Surveillance and remote monitoring cameras are often operating with acertain delay, configured at implementation or at camera installation,in order to optimize bitrate and quality. Further, surveillance camerasare often operating with a fixed quality, i.e. resolution and resultingencoding target bitrate, which is also configured at deviceimplementation, hard coded, or at device installation. For remote-drivenor autonomous driving vehicles, there is sometimes the need to interactwith the passengers and therefore the need to turn the camera system forsurveillance into unidirectional conversational mode. As a result, thenetwork and content jitter buffers should be reduced and optionally thecodec configuration may be changed. A codec configuration forconversational often operates with different encoder features and alsowith lower bitrates to reduce latency. There is currently no directlatency control solution, which modifies the buffer level on the senderand the receiver side, while the stream is active.

SUMMARY

It would be advantageous to provide for a monitoring and surveillancesystem that is able to be adequately used for monitoring and forsurveillance purposes.

It would also be advantageous to provide for method that is able toadequately operate a monitoring and surveillance system.

In a first aspect, there is provided a monitoring and surveillancesystem arranged for processing video data associated with a vehicle,wherein said system is arranged to operate in at least two operatingmodi, a first modus of said two modi being associated with a firstlatency requirement for said video data and a second modus of said twomodi being associated with a second latency requirement.

The system comprising:

-   -   a camera unit, arranged to be installed in said vehicle, wherein        said camera unit is arranged for capturing video data;    -   a streaming unit, arranged to be installed in said vehicle, and        arranged for receiving said video data and for transmitting said        video data over a telecommunication network to a video        processing server;    -   said video processing server arranged for selecting a modus of        said at least two operating modi, and for communicating said        selected modus, over said telecommunication network, to said        camera unit such that said streaming unit can be tuned to said        selected modus.

The inventors have found that the monitoring and surveillance systemshould be able to cope with at least two modi, wherein each of the modusrequired a different latency setting.

A first modus may, for example, be directed to the monitoring aspect anda second modus may, for example, be directed to the surveillance aspect.In any case, the latency requirement of the first modus differs from thelatency aspect of the second modus.

In accordance with the present disclosure, the video processing serverselects a modus to be used for the monitoring and surveillance system.That is, the video processing server determines, i.e. selects, in whichof the at least two modi the system should be operative.

Once a selection is made, that particular selection is to becommunicated to the camera unit, such that the streaming unit can betuned to the selected modus. The streaming unit needs, for example, toknow what the latency requirement is in order to make sure that videopackets are timely transmitted, over the telecommunication network, tothe video processing server.

In accordance with the present disclosure, the camera unit and thestreaming unit are deployed in a vehicle, for example a train, drone,aircraft, or anything alike. The video captured by the camera unit maybe stored locally, i.e. without communicating the video data to theoutside world. Alternatively, or in addition to, storing the video datalocally, the streaming unit may transmit the video data to a videoprocessing server which is accessible over the telecommunicationnetwork. The video processing server may be located inside thetelecommunication network, or, for example, somewhere in the internet.In accordance with the present disclosure, the video processing serveris reachable via the telecommunication network, for example via a ThirdGeneration, 3G, connection, a Fourth Generation, 4G, connection or evena Fifth Generation, 5G, connection.

It is noted that the monitoring and surveillance system is arranged forprocessing video data associated with a vehicle. The monitoring andsurveillance system may, alternatively, also be used for processingvideo data associated with non-vehicles. For example, the monitoring andsurveillance system may be arranged for processing video data associatedwith a particular venue, a festival, a building, an office, a particularlocation, a park or anything alike.

In an example, the streaming unit comprises a transmitting queuearranged for queuing video data packets to be transmitted to said videoprocessing server, and wherein said video processing server comprises areceiving queue arranged for queuing video data packets received fromsaid streaming unit.

The inventors have found that the timing delay may be composed ofseveral components. One of the components is directed to thetransmitting queue, and the amount of packets already residing in thetransmitting queue. Another component is directed to the receiving queueand the amount of packets residing in the receiving queue. Yet anothercomponent is directed to the communication time, i.e. the time requiredto send the packets from the streaming unit to the video processingunit. The latter may also encompass the time delay encountered by thewireless communication channel, as well as all kinds of MAC layerprotocol delays.

In a further example, the video processing server is further arranged toswitch from said first modus to said second modus, and wherein saidstreaming unit and said video processing server are arranged fordiscarding video data packets in their respective queue which are olderthan said second latency requirement. The inventors have found asolution in which the system decides to switch from the first modus tothe second modus, more particularly to a modus in which there is a firstlatency to a modus in which there is a second latency requirement,wherein the second latency requirement is stricter compared to the firstlatency requirement.

The solution is that all packets in the respective may be assigned aparticular timing parameters. The timing parameter relates to theposition of the video packet in the queue as well as the communicationtime. Packets having a timing parameter older than the second latencyrequirement may, subsequently, be discarded when switching from thefirst modus to the second modus.

In a further example, the streaming unit is arranged for determining adue deadline for transmitting a video data packet in said transmittingqueue to said video processing server based on at least one of:

-   -   a network jitter buffer duration;    -   frame durations of preceding video data packets present in said        transmitting queue.

In yet another example, the system comprises a plurality of camera unitsand a single streaming unit, wherein each of said plurality of cameraunits is arranged to provide video data to said single streaming unit.

The inventors have found that, typically, in a vehicle multiple cameraunits are deployed. For example, in train wagons often two camera unitsare deployed. The amount of camera units then relates to the amount oftrain wagons. Only one streaming unit may be deployed, wherein that onestreaming unit is receiving the video data from all of the camera unitsseparately.

In a second aspect, there is provided an in-vehicle camera assemblyarranged for capturing video data with respect to monitoring andsurveillance, wherein said in-vehicle assembly is arranged to operate inat least two operating modi, a first modus of said two modi beingassociated with a first latency requirement for said video data and asecond modus of said two modi being associated with a second latencyrequirement.

The assembly comprising:

-   -   a camera unit, arranged to be installed in said vehicle, wherein        said camera unit is arranged for capturing video data;    -   a streaming unit, arranged to be installed in said vehicle, and        arranged for receiving said video data and for transmitting said        video data over a telecommunication network to a video        processing server;

wherein said camera unit is further arranged for receiving a selectedmodus, from said video processing server, over said telecommunicationnetwork, and for tuning said streaming unit based on said receivedmodus.

The advantages of the first aspect of the disclosure are also inherentlya part of the second aspect, and third aspect, of the disclosure.Furthermore, it is pointed out that although the claims read as if allthe modules/equipment/units according to this second aspect of thepresent disclosure are incorporated into a device, a person skilled inthe art understands that the same disclosure could be implemented by,for example, distributing each of the modules/equipment/units overmultiple devices.

Further, it is noted that the equipment may also be referred to as amodule, unit, device or anything alike.

In a further example, the streaming unit comprises a transmitting queuearranged for queuing video data packets to be transmitted to said videoprocessing server.

In yet another example, the received modus is said second modus suchthat said modus switches from said first modus to said second modus,wherein said streaming unit is arranged for discarding video datapackets in said transmitting queue which are older than said secondlatency requirement.

In an even further example, the streaming unit is arranged fordetermining a due deadline for transmitting a video data packet in saidtransmitting queue to said video processing server based on at least oneof:

-   -   a network jitter buffer duration;    -   frame durations of preceding video data packets present in said        transmitting queue.

In an example, the assembly comprises a plurality of camera units and asingle streaming unit, wherein each of said plurality of camera units isarranged to provide video data to said single streaming unit.

In a third aspect of the present disclosure, there is provided a methodof operating a monitoring and surveillance system in accordance with anyof the examples as provided above, wherein said method comprises thesteps of:

-   -   capturing, by said camera unit installed in said vehicle, video        data;    -   receiving, by said streaming unit installed in said vehicle,        said video data and transmitting said video data over said        telecommunication network to said video processing server;    -   selecting, by said video processing server, a modus of said at        least two operating modi, and communicating said selected modus,        over said telecommunication network, to said camera unit such        that said streaming unit can be tuned to said selected modus.

In an example, the method comprises the steps of:

-   -   switching, by said video processing server, from said first        modus to said second modus, and    -   discarding, by said streaming unit and by said video processing        server, video data packets in their respective queue which are        older than said second latency requirement.

In a fourth aspect of the present disclosure, there is provided a methodfor capturing video data with respect to monitoring and surveillanceusing an in-vehicle camera assembly in accordance with any of theexamples as provided above, wherein said method comprises the steps of:

-   -   capturing, by said camera unit, video data;    -   receiving, by said streaming unit, said video data and        transmitting, by said streaming unit, said video data over a        telecommunication network to a video processing server;    -   receiving, by said camera unit, a selected modus, from said        video processing server, over said telecommunication network,        and    -   tuning said streaming unit based on said received modus.

In a fifth aspect of the present disclosure, there is provided acomputer program product comprising a computer readable medium havinginstructions which, when executed by a monitoring and surveillancesystem cause said system to implement a method in accordance with any ofexamples as provided above.

The above mentioned and other features and advantages of the disclosurewill be best understood from the following description referring to theattached drawings. In the drawings, like reference numerals denoteidentical parts or parts performing an identical or comparable functionor operation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 discloses an example of a vehicle having a front facing camera aswell as in wagon camera's.

FIG. 2 illustrates a system according to the present disclosure.

FIG. 3 illustrates a system according to the present disclosure.

FIG. 4 illustrates a method according to the present disclosure.

FIG. 5 illustrates a method according to the present disclosure.

DETAILED DESCRIPTION

FIG. 1 discloses an example of a vehicle 1, more particularly a train 3,having a front facing camera as well as in wagon cameras.

The front facing camera has a viewing angle as indicated with referencenumeral 4. The front facing camera may aid in driver assistancefunctions, such as obstacle on track detection or the like.

The train may also have one or more surveillance cameras, as depictedusing reference numeral 2, which are mounted inside the wagons of thetrain. The surveillance camera's thus monitor the inside of wagons. Theviewing angle is, schematically, indicated with reference numeral 5.

The trains as indicated with reference numeral 3 may be combined into asingle train, such that the single train comprises the front facingcamera as well as in wagon cameras.

FIG. 2 illustrates a system 10 according to the present disclosure. Thevarious CCTV cameras 11 are all connected to an In-Train platform 12,which is recording all camera feeds on a local storage system such ashard disks, for example. As a default, the in-train platform 12 may notstream any video camera into the operation center 15.

The operation center 15 can interact with the in-train platform 12 via acontrol connection 13. The control connection 13 may use any suitableradio interface 14. The operation center 15 may select, which camerashould be up streamed, by giving instructions such as streaming start,streaming stop per camera. Such streaming may be performed optionally inaddition to the local recording of the video data. The operation center15 can also control the video quality and also the content and networkjitter buffering using internal modules such as for video ingest 16 andrendering & control 17.

In FIG. 3, reference numeral 30 illustrates a more elaborateillustration of the system according to the present disclosure. Thecamera 32 contains the actual video sensor 34. The sensor may be, forexample, a Charge Coupled Device, CCD or a Complementary Metal OxideSemiconductor, CMOS sensor. The camera 32 further comprise some optionalvideo pre-processing 35, for example to remove lens distortion, and alsothe video encoding and packaging 36 and the streaming function 37, 38.The streamer function 37, 38 contains an in-queue 37, which store thecompressed video frame until time of upstream. Reference numeral 31refers to an input of video frames captured by the camera 32 and thenumeral 33 refers to a source configuration and control module.

The Ingest server 42 handles the reception and contains also an outqueue 45. The Network and Content jitter buffer management is donebetween the video encoding and packaging function 36 and the videoprocessing function 47. The encoder 36 produces compressed video frameswith a certain, fixed or varying, frame rate and puts the frames intothe in-queue 37. On the video ingest side 42, the received compressedframes are first put into the out-queue 45. The ingest function 42forwards the video frames to the next function—video processing 47.There is a fixed delay between in-queue 37 and out-queue 45. However,the frames might be queued either on the in-queue side or on the outqueue side, depending on the current channel condition—radio link 40between camera 32 and ingest 42. As mentioned earlier, the operationcenter 41 interacts with the cameras 32 on the train using a controlconnection 39. The operation center 41 may internally comprise ofadditional modules such as a sink control 43 and a controller 46.

The encoder 36 spits out encoded frames with a certain time interval,which is generally equal to the frame rate, and enqueues the frames Inthe in-queue 37. The radio channel 40 may be busy for some time so thestreamer may not transmit frames according to the arrival rate. Thestreamer function knows the configured content and network jitter bufferduration and can hold back frames up to a due date. The reception of theframes may be delayed accordingly. The out queue 45 is compensating thereception jitter. The received frames sit in the out queue 45 up to acertain due date. The due date of a frame is calculated relative to anabsolute receiver Reference time. For example, the out due date iscalculated using the absolute receiver Reference time plus the networkjitter buffer duration and the sum of the preceding frame durations.

When the controller changes from surveillance mode, with a long networkand content jitter buffer duration of for example 2 seconds, to aconversational mode having a short jitter buffer of for example 50 ms,the In Queue 37 and the Out Queue 45 need to discard a set of framescorresponding to the 2000 ms-50 ms time duration. The Out Queue 45manager is calculating the new due date target and discarding any frame,which is already older than the new due date. The same procedure occurson the In Queue 37 side, where the in queue manager is removing anyframe, which is expired already.

The ingest server 42 needs to determine the amount of discarded frameson the in queue 37 side in order to calculate the new due datecorrectly. The out queue manager can use the progress of the encodingtimestamps or presentation timestamps or a new box.

The Source Config & Control function 33 fetches the configurations fromthe Sink Control function 43 using a Hypertext Transfer Protocol, HTTP,GetConfiguration command. After retrieval of the configuration, theSource Config & Control 33 parses the received configuration data andfinds the protocol identifier and Uniform Resource Locator, URL, for thecontrol connection. When a web socket is used, the protocol may beidentified by the scheme “ws://”. The Source Configuration and Controlfunction 33 establishes the Web Socket connection with the Sink Controlfunction 43 and maintains the connection using Keep alive mechanisms.

When at some later point in time the Controller 46 desires to toggle thelatency to live latency, the Controller 46 sends a command to the SinkController 43, which then modifies the Out Queue 45 handling and alsosends a Reduce In Queue Buffer duration to the Framework for Live UplinkStreaming, FLUS, Source via the established control connection.

The Camera 32 and the Encoder 36 are continuously capturing and encodingframes. The framerate may be variable, depending on the capturecapability. The Encoder 36 is inserting encoded frames into the In Queue37 according to the frame duration (i.e. variable or fixed frame rate).This process is generally not impacted by the latency modificationcontrol command.

The streamer is taking encoded frames out of the In Queue 37 and sendingthe frames via the transmission channel 40 according to the bitrate ofthe channel 40. The transmission duration may be shorter or larger thanthe frame duration.

The video ingest 42 is receiving frames from the channel 40 andinserting each frame into the Out Queue 45. The intention of the outqueue 45 is to de-jitter and to recover the original frame rate i.e. therate (fixed or variable) at which the encoder 36 has pushed coded framesinto the In Queue 37. For this process, a due date is calculated foreach frame, depending on the buffer duration and the progress ofpresentation times, wherein the presentation time of a frame iscalculated as sum of all previous frame durations assuming here variableframe durations.

When the controller 46 instructs to toggle from surveillance to livemode, the controller 46 sends a command to the Sink Controller 43, whichis then influencing the procedure to take frames from the In Queue 37and also when to forward frames from Out Queue 45 to Process function47.

The Out Queue Manager goes through the list of enqueued frames andcalculates a new due date. The new due date is calculated based on thenew jitter buffer duration from the controlled. The Out Queue managerdiscards any frame, which should have already been sent according to thenew due date.

When the media bitrate is marginally higher than the link bitrate, theFLUS Source may not able to upstream frames according to arrival time(spitted out from the encoder) and frame duration. In this case, theparts of the frames, worth of 5 sec buffer depth, are still located inthe FLUS Source buffer, while the rest is already uploaded.

The method 50 in FIG. 4 comprises the steps of capturing 51 video data,by a camera unit installed in a vehicle. In a further step of receiving52, the streaming unit installed in the vehicle, receives the video dataand also transmits the video data over a telecommunication network to avideo processing server.

In a step of selecting 53, the video processing server selects a modusfrom at least two operating modi, and communicates the selected modus,over the telecommunication network, to the camera unit such that thestreaming unit can be tuned to the selected modus.

The method 50 may further comprises the step of switching 54 the videoprocessor, from a first modus to a second modus, and discarding 55, bythe streaming unit and by the video processing server, video datapackets in their respective queue which are older than said secondlatency requirement.

The method 60 in FIG. 5 illustrates a method for capturing video datawith respect to monitoring and surveillance using an in-vehicle cameraassembly. The method 60 comprises the steps of capturing 61 video databy the camera unit. IN a further step of receiving 62 the streaming unitreceives the video data and transmits, the video data over atelecommunication network to a video processing server. In a furtherstep of receiving 63, the camera unit receives a selected modus, from avideo processing server, over a telecommunication network, and thestream unit tunes 64 the streaming unit based on the received modus.

Other variations to the disclosed examples can be understood andeffected by those skilled in the art of practicing the claimeddisclosure, from a study of the drawings, the disclosure, and theappended claims. In the claims, the word “comprising” does not excludeother elements or steps, and the indefinite article “a” or “an” does notexclude a plurality. A single processor or other unit may fulfil thefunctions of several items recited in the claims. The mere fact thatcertain measures are recited in mutually different dependent claims doesnot indicate that a combination of these measures cannot be used toadvantage. A computer program may be stored/distributed on a suitablemedium, such as an optical storage medium or a solid-state mediumsupplied together with or as part of other hardware, but may also bedistributed in other forms, such as via the Internet or other wired orwireless telecommunication systems. Any reference signs in the claimsshould not be construed as limiting the scope thereof.

The present disclosure is not limited to the examples as disclosedabove, can be modified and enhanced by those skilled in the art beyondthe scope of the present disclosure as disclosed in the appended claimswithout having to apply inventive skills.

1.-14. (canceled)
 15. A monitoring and surveillance system arranged for processing video data associated with a vehicle, said system being arranged to operate in at least first and second modes associated with respective first and second latency requirements for said video data, said system comprising: a camera assembly arranged for installation in said vehicle and comprising: a camera arranged to capture video data; a streaming unit arranged to receive said captured video data and to transmit said captured video data over a telecommunication network to a video processing server; and a source controller arranged to: receive a selected mode, of at least the first and second modes, from said video processing server via said telecommunication network, and tune said streaming unit to operate according to said received selected mode; and the video processing server, being arranged to: select a mode from at least the first and second modes, and communicate said selected mode, over said telecommunication network, to said source controller.
 16. The system of claim 15, wherein: said streaming unit comprises a transmitting queue arranged for queuing video data packets to be transmitted to said video processing server, and said video processing server comprises a receiving queue arranged for queuing video data packets received from said streaming unit.
 17. The system of claim 16, wherein: said video processing server is further arranged to: select a mode by switching from said first mode to said second mode, communicate the selected second mode to said source controller, and subsequently discard video data packets in the receiving queue which are older than the second latency requirement; and said streaming unit is further arranged to discard video data packets in the transmitting queue which are older than the second latency requirement, in response to the source controller receiving said selected second mode.
 18. The system of claim 17, wherein said streaming unit is further arranged to determine a due deadline for transmitting a video data packet in said transmitting queue to said video processing server based on at least one of: a network jitter buffer duration; and frame durations of preceding video data packets present in said transmitting queue.
 19. The system of claim 15, wherein: the camera is one of a plurality of cameras included in said system, and the plurality of cameras are arranged to provide captured video data to said streaming unit.
 20. A camera assembly arranged for in-vehicle installation and for capturing video data with respect to monitoring and surveillance, said camera assembly being arranged to operate in at least first and second modes associated with respective first and second latency requirements for said video data, said camera assembly comprising: a camera arranged to capture video data; a streaming unit arranged to receive said captured video data and to transmit said captured video data over a telecommunication network to a video processing server; and a source controller arranged to: receive a selected mode, of at least the first and second modes, from said video processing server via said telecommunication network, and tune said streaming unit to operate according to said received selected mode.
 21. The camera assembly of claim 20, wherein said streaming unit comprises a transmitting queue arranged for queuing video data packets to be transmitted to said video processing server.
 22. The camera assembly of claim 21, wherein said streaming unit is further arranged to, when the received selected mode indicates a mode switch from the first mode to the second mode, discard video data packets in the transmitting queue which are older than the second latency requirement.
 23. The camera assembly of claim 22, wherein said streaming unit is arranged to determine a due deadline for transmitting a video data packet in said transmitting queue to said video processing server based on at least one of: a network jitter buffer duration; and frame durations of preceding video data packets present in said transmitting queue.
 24. The camera assembly of claim 20, wherein: the camera is one of a plurality of cameras included in said camera assembly, and the plurality of cameras are arranged to provide captured video data to said streaming unit.
 25. A method for capturing video data with respect to monitoring and surveillance using an in-vehicle camera assembly comprising a camera, a streaming unit, and a source controller, the method comprising: capturing video data by said camera unit; receiving, by said streaming unit, said captured video data; transmitting, by said streaming unit, said captured video data over a telecommunication network to a video processing server; receiving, by said source controller, a selected mode from said video processing server via said telecommunication network, wherein the selected mode is one of at least first and second modes associated with respective first and second latency requirements for said video data; and tuning, by said source controller, said streaming unit to operate according to said received selected mode.
 26. The method of claim 25, further comprising when the received selected mode indicates a mode switch from the first mode to the second mode, discarding video data packets in a transmitting queue of the streaming unit which are older than the second latency requirement.
 27. The method of claim 26, further comprising when the received selected mode indicates a mode switch from the first mode to the second mode, discarding video data packets in a receiving queue of said video processing server which are older than the second latency requirement.
 28. The method of claim 26, further comprising determining, by said streaming unit, a due deadline for transmitting a video data packet in said transmitting queue to said video processing server based on at least one of: a network jitter buffer duration; and frame durations of preceding video data packets present in said transmitting queue.
 29. A non-transitory, computer-readable medium having instructions that, when executed by an in-vehicle camera assembly arranged to capture video data with respect to monitoring and surveillance, configure the in-vehicle camera assembly to perform operations corresponding to the method of claim
 25. 